bench: community report, 4x RTX 3090 layer split, stock 0.1.44 against 0.1.43 with a patch series, 1 to 16 users by havok2-htwo · Pull Request #2364 · Niko1221/Strata

havok2-htwo · 2026-10-14T03:18:23Z

This PR adds results only: the report folder
bench/results/2026-10-14-community-4x-rtx-3090-0144/
and one bullet in docs/COMMUNITY_BENCHMARKS.md. It changes no code.

Hardware: 4x RTX 3090 24 GB at a 380 W limit each. PCIe: two cards at Gen3 x16, two at Gen4 x8. The rest of
the machine is a Threadripper 3990X (no AVX-512) with 251 GiB RAM and an NVMe RAID0, running Ubuntu 24.04,
driver 580.178.04 and CUDA 13.0.

Model: Unsloth UD-IQ4_XS (shards match the pins in setup.py), with the native-experts pack and the MTP draft
made by setup. The context is 262,144 tokens with --kv int8 --kv-resident 32768, the layer split is 12,25,37,
and all experts are resident in VRAM. Text only.

Configs (same day, same engine arguments, each build with its own serve/server.py):

stock: v0.1.44 ed4773eb, built from source with setup.py's CUDA flags. It runs 8 slots in 4 groups of 2.
8 is the most 0.1.44 runs on this split with its default pipelined groups (without --batch-mtp); with
--batch-mtp and serial windows (--batch-groups 1 or STRATA_BATCH_MTP_SERIAL=1) it accepts more, which we
did not test. v0.1.44.1 (296b59d2) came out a few hours before the run and was not measured.
modified: 0.1.43 plus our unmerged patch series, built with -DSTRATA_MMQ_TAB=ON. It runs 16 slots in 4
groups of 4, with the opt-in switches listed in the README. This is the build we run in production. Source,
pinned: havok2-htwo/Strata at 08830ddf;
the same commits as patches 0001-0052 in
havok2-htwo/StrataQ38FN at 4ece0ca
plus the last commit 08830ddf (server only).

Aggregate output tok/s (stock / modified), 1 run per arm:

Workload	stock 0.1.44	modified 0.1.43
1 user, ~147-token prompt, 512 out	111.7	116.2
8 users, same	314.4	399.4
16 users, same	292.9 (TTFT p50 8.69 s)	516.7 (0.89 s)
16 users, ~1.2K-token prompt	238.1 (TTFT p50 6.96 s)	406.5 (1.91 s)
16 agents x 3 turns, temperature 0.7	92.0 (follow-up TTFT p50 9.91 s)	151.9 (1.52 s)

The greedy replies were byte-identical between the two builds for all 82 sweep requests. The README also
gives the engine's own timings:

prompt read rates
the admission copy into a slot, as the median per phase: 153-197 ms on stock, 1.0-13.2 ms on the modified build
solo MTP acceptance in the 1-user level, identical in both builds (586 of 948, 61.8 %)
a note on [0.1.43] CUDA layer split (2× RTX 3090): decode windows lose async launch, solo 131.4 → 39–76 tok/s, prefill −35% #2038: every solo request drafted in both builds, and a request admitted into a batch slot logs one
strata serve: prompt line with 1 generated token and 0 of 0 drafts

Limitations:

The 16-user rows mix 16 against 8 slots with everything else that differs. The 8-user row is the closer
comparison, and there both builds carry 2 rows per group-step. Even there the gap mixes the patches and switches,
the -DSTRATA_MMQ_TAB=ON build option and the 0.1.43 against 0.1.44 base.
The modified build used about 26 GiB more system RAM after load (96.2 against 70.4 GiB, whole machine).
No row isolates a single patch.
There is one run per arm, in one order (modified first).
Images, prompts above 20K tokens, more than 16 users, --batch-mtp (also stock above 8 slots in serial
windows) and v0.1.44.1 are untested.

Two parts of the series (more than 8 slots on a layer split, and the device-to-device slot copy) would come as
separate PRs, each with its own A/B. This report does not measure either alone.

🤖 Generated with Claude Code

…t 0.1.43 with a patch series, 1 to 16 users Same-day serving comparison on one machine (4x RTX 3090 24 GB, layer split 12,25,37, Threadripper 3990X, 251 GiB RAM), Unsloth UD-IQ4_XS, 262,144-token context, text only, one run per arm: stock v0.1.44 (ed4773e, source build, 8 slots, the most it runs on this split with its default pipelined groups, without --batch-mtp) against a modified 0.1.43 with an unmerged patch series (16 slots, opt-in switches, -DSTRATA_MMQ_TAB=ON; source pinned at havok2-htwo/Strata 08830dd), each through its own serve/server.py. Aggregate output tok/s, stock / modified: 111.7 / 116.2 at 1 user, 314.4 / 399.4 at 8, 292.9 / 516.7 at 16 (short prompts), 238.1 / 406.5 at 16 with 1.2K prompts, 92.0 / 151.9 for 16 agents x 3 turns. Greedy replies byte-identical between the builds (82/82). The modified build used about 26 GiB more system RAM. v0.1.44.1 and --batch-mtp were not measured. Sanitized per-level rows, per-request reply hashes, configs, engine log excerpts, GPU samples and the scripts used. Results only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ssion and decode switches, and the 4x R9700 numbers (PR #2364 port)

…stream's new trees; the engine stays this branch's Upstream's 264 commits since 296b59d. Where both sides changed a file of the engine (src/, include/: 39 files) or CMakeLists.txt, this branch's version stays whole, and weights.cpp too (upstream's adds only STRATA_HC_REQ8's loader, which needs its header; --hc-q8 does this here). The files only upstream changed are upstream's: the GGUF reader (an all-zero compress_ratios list is a converter's placeholder, Niko1221#2396), the BF16 dequantization (the wide kernels now on by default on sm_86 too: the prompt path's dense matrices on the 3090, bitwise; prompt times unchanged), the AVX2 expert rows' exact one-token form and STRATA_PRIMARY_DEVICE's helpers (neither called here). The engine keeps its version, 0.1.43.1. Taken: - serve/: a request without sampling fields gets the model card's set for its thinking state under the config's block (Niko1221#2685; this branch's /v1/choose still reads the model's own logits); max_tokens counts the thinking (the answer keeps a quarter; "answer_room": false turns it off); Anthropic's thinking.budget_tokens is a budget too; a prompt is encoded from the last special token it shares with a recent one (Niko1221#567); a tool body is not rescanned before a closing tag; logit_bias lists normalized once; the Prometheus gauges over the window between scrapes (Niko1221#2341); a config's joined args split and its JSON errors placed (Niko1221#2267); an explicit reasoning false kept; the batch slots' turn checkpoint pick (Niko1221#2518) and STRATA_ADMIT_SHORT_FIRST. - web/: the page sends no sampling until it is changed. - setup.py, tools/, docs/, bench/results, the Metal, SYCL, HIP and Qwen3.6 trees and their tests. test_parser_call_scan declares a tool: without one this branch's parser passes the text through verbatim. Not taken (the engine files they change stay this branch's; upstream's new sources are in the tree, not built): - The cuda86 group (5a6f4ed): STRATA_QSA_ROWS, STRATA_ATTN_DAILY, STRATA_QSA_SPLIT_ATTN, STRATA_SIDE_MIXER (and its fix 477ac13), STRATA_HC_REQ8, STRATA_PROMPT_ATTN_IMMA, the grouped short-prompt gather and the cooperative top-k digit: this engine's own kernels do these (SPLIT_ATTN and SIDE_MIXER are ports of its fd23d91 and 34a35db); STRATA_B6_MMVQ_ROWS (that kernel is 1.3-3.8% of the 3090's time here); STRATA_EXPDEC, STRATA_HC_UPMIX and the once-per-token quantization of the MoE input (c675f5b): open items. 7e8485a: every KV format takes the batched window here. - The cpu group: the AVX2 router and Q5_1/Q8_0 row dots (the router runs on the GPU here, STRATA_KQ256 is off), STRATA_CPU_IQ_ALL_EXACT1 (CPUs without AVX-512), --host-core sibling and --pool-affinity measure (hybrid CPUs), the --lookup-chain cap (no such option here). - The cache group (the resident RAM mode's options, STRATA_PCIE_BALANCE, the tier's evictions under NOWAIT, the WSL K/V host copy) and fix3 (the PLE gather_batch fix Niko1221#2622: no such fast path here; the layer split's rules; the P2P probe Niko1221#2666: no peer copies here): code this engine does not have. - The conversation cache's reserve, park gain, pinned prefixes and borrowing: upstream's conversation cache is not built here (this engine has its own prompt cache; --prompt-cache-spread's and the borrowing's ideas are open items). - havok (Niko1221#2364), batch, batch-1441: batch slots, layer splits, reads beside decoding slots (one request at a time here). The pointer-table MMQ (STRATA_MMQ_TAB, STRATA_MMQ_DIRECT) and STRATA_MMQ_GROUP_SORT: open items. - sm75, rdna2, rdna34, sycl, Metal: other hardware (Niko1221#2072's split top-k and halving scorer and 6befb3d's Q6_K form: open items). The Qwen3.6 engine. Window hashes on fixed text (thinking, 1,200 tokens, --adapt-every 0, the second GPU's tier pinned) equal the previous build's on both models; the server's 862 unit tests pass (10 skipped); api_cache_test and api_stream_test pass on both models.

havok2-htwo · 2026-10-18T06:43:59Z

Thank you for porting the series into 0.1.45 and for the R9700 numbers. I have two follow-ups from a run on stock 0.1.45 (477ac13c, built with setup.py's release flags, CUDA 13.0) on 4x RTX 3090 24 GB (PCIe). The model is Qwen3.8-Flash-Next UD-IQ4_XS with the native pack, run with --layer-split 12,25,37 --trim-stage-weights --resident-experts --prefill 512 --spec 4 --kv int8 --kv-resident 28672, 262K context, greedy.

STRATA_SLOT_CKPT_MOVE=1 and tools/batch_groups_cache_test.py. I ran the test unchanged. All arms had the other merged switches on, and they differ only in MOVE and CKPT_POOL. One run per arm:

	merged switches, no MOVE	+ MOVE, no `CKPT_POOL`	+ MOVE + `CKPT_POOL`
U (`--batch 2 --batch-groups 2`, the docstring's args)	PASS	PASS	FAIL
B16P (`--batch 16`, same cache args)	PASS	FAIL	FAIL

In each of the three FAIL runs, the request list differs from the PASS runs by a single extra line in round 1, and that line comes from how the test counts:

PASS: (3141, 0), (3142, 0), (3245, 3220), (3246, 3245), (3246, 3221), ...
FAIL: (3141, 0), (3142, 3141), (3142, 0), (3245, 3220), ...

In the FAIL runs, conversation A started alone on the solo path. When B arrived, A was moved into a batch slot (StrataEngine.generate_batched in serve/server.py, prompt = list(ids) + out # promoted: it continues in a batch slot from here), and the engine logged the same request a second time as prompt + first token: 3142 tokens = 3141 reused + 1 read. The test assumes round 1 is exactly two lines (reqs[2:]). With the extra line, B's real cold read (3142, 0) falls into the checked part, and the test fails. The real round-2/3 requests reuse 99.2 % or more in all six stock runs. The answers are identical round by round between every MOVE-on and MOVE-off pair.

Whether A starts solo in round 1 looks like the arrival race between the test's two threads. In the PASS runs the log has slot 0 takes 3141 tokens before A's prompt line. In the FAIL runs A's solo read ends first and slot 0 takes 3142 tokens follows. It happened in 3 of the 4 runs with MOVE and in 0 of the 2 without. With one run per arm I can't rule out that MOVE makes the solo start more likely. The last line of each run, (3375, 3374), is the same kind of re-log in the other direction: slot back to solo (#857).

Suggested test fix: count one line per request, the read itself, and skip these continuation lines. A request that moves between the solo path and a slot is logged again with its prompt plus the tokens it generated so far, all of them reused but the last one (reused = prompt - 1). No real follow-up in this test reads only one token, since each new turn adds about 25 tokens:

reqs = [(p, r) for p, r in reqs if r != p - 1]   # drop solo<->slot continuation lines of the same request
ok = len(reqs) == 6                              # was >= 6

I have not re-run the changed test. Applied to the saved engine logs of these runs, the changed check passes all six stock runs and six runs of our fork's production build (MOVE on and off, three cache setups). Those six fail the current count with MOVE on and off alike: in our build A started solo in all six, so the same kind of continuation lines appear. I can send this as a small PR if you want it. Whether MOVE then deserves more than "experimental" is your call. This test is all I checked it with, and in every run it reports conversation cache restores: 0: conversations were parked but never restored (the reuse came from the slots giving back), so it never reaches the parked-conversation case your BATCHING.md row warns about.

The merged switches on CUDA. I used the same build and arguments, with a 128 GiB / 448-slot conversation cache (--conversation-cache-mib 131072 --conversation-cache-slots 448) and STRATA_SPLIT_MTP_BATCH=1 STRATA_KV_PREFETCH=1 STRATA_NAN_GUARD=0 in all arms. There was one round per arm, run in the order shown, not your 3 interleaved rounds. The switches arm has every merged switch except MOVE: SLOT_COPY_DIRECT, SLOT_COPY_BATCHSYNC, CKPT_POOL, CKPT_THREADS, ADMIT_MERGE_FIRST, PIPE_BYIELD_NODRAIN, BATCH_SELF_COMMIT, QSA_SLOTS, HANDOFF_PIPE=8, MMQ_GROUP_SORT, and the server's ADMIT_SHORT_FIRST=8192. The load comes from our own scripts. bench_concurrency.py sends a ~147-token (143-152 tokens) or ~1.2K-token (1210-1217) prompt with 512 tokens out and 2 requests per user. bench_multiturn.py runs 16 users x 3 turns over an 8K document. Output tok/s, with TTFT p50 in brackets:

	8 slots	16 slots	16 slots + switches
1 user, short	112.5 (0.34 s)	112.6 (0.34 s)	115.2 (0.26 s)
8 users, short	317.7 (1.84 s)	317.8 (1.61 s)	371.1 (1.17 s)
16 users, short	315.8 (13.02 s)	377.8 (1.87 s)	467.8 (1.01 s)
16 users, 1.2K prompts	240.1 (17.95 s)	276.4 (3.92 s)	335.8 (3.48 s)
16 users x 3 turns, follow-up TTFT	95.9 (13.88 s)	99.3 (3.03 s)	121.8 (2.30 s)

Against your R9700 table (0.1.44.1, IQ3_S, HIP; here 0.1.45, UD-IQ4_XS, CUDA), 16 slots + switches over 8 slots:

Measure	4x RTX 3090	R9700
16 users, short	1.48x	1.56x
16 users, 1.2K prompts	1.40x	1.41x
8 users	1.17x	1.20x
16 x 3 turns, output tok/s	1.27x	1.12x
16 x 3 turns, follow-up TTFT	13.88 to 2.30 s	21.2 to 5.7 s
Share of the 16-user gain from the slot count alone	41 %	38 % (BATCHING.md table; the release notes say about 40 %)

For the single-turn rows the ratios are close. The multi-turn gain is larger here.

Reply identity:

Short prompts: all 50 replies are identical between 8 and 16 slots, and between 16 slots and 16 slots + switches.
1.2K prompts: 31/32 identical between 8 and 16 slots (one 512-token reply differs) and 32/32 with the switches.
Multi-turn (48 replies per arm): turn 1 is 16/16 identical in both pairs. Turn 2 is 12/16 in both pairs, but in other users (10, 11, 13, 15 for 8 vs 16 slots; 8, 13, 14, 15 for 16 slots vs + switches). Turn 3 is 9/16 and 7/16, 37/48 and 35/48 in total. The 8 vs 16 slot pair has no switch set, so the multi-turn path is not exact here even without the switches. I did not chase which differences, if any, the switches cause.

In this release build STRATA_MMQ_DIRECT=1 does nothing. The engine says STRATA_MMQ_DIRECT=1 is ignored: this build has no pointer-table MMQ (CUDA with STRATA_MMQ_TAB only), as the release notes say. STRATA_MMQ_GROUP_SORT=1 did apply without the tab (prompt MMQ sorts the full expert groups (STRATA_MMQ_GROUP_SORT=1): 48 of 48 layers), so the BATCHING.md row that says both need -DSTRATA_MMQ_TAB=ON is only true of MMQ_DIRECT.

Not tested: a -DSTRATA_MMQ_TAB=ON build of 0.1.45, more than one round per arm, sampled output, a soak, other models, HIP/SYCL/Metal.

The PR itself only carries the 0.1.44 against 0.1.43 + series comparison, which is out of date now that the code is in 0.1.45. Should I close it?

Niko1221 · 2026-10-19T17:19:21Z

Thanks for the pull request.

We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

To reopen it takes about two minutes:

Rebase on main if it no longer applies, and re-run your checks on 0.2.0-beta.
Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
Fill in the pull request template: the problem and result, and the validation (what you ran, on which revision, what you saw).
Comment /reopen. A check runs and either reopens the pull request or lists what is still missing.

The rules are in CONTRIBUTING.md.

Niko1221 added a commit that referenced this pull request Oct 17, 2026

docs/BATCHING.md: more than 8 slots on a layer split, the opt-in admi…

79e5cd1

…ssion and decode switches, and the 4x R9700 numbers (PR #2364 port)

Niko1221 added a commit that referenced this pull request Oct 17, 2026

Merge f-145-havok (#2364 multi-user series, opt-in)

df44bcb

Niko1221 closed this Oct 19, 2026

Navigation Menu

Repository navigation

bench: community report, 4x RTX 3090 layer split, stock 0.1.44 against 0.1.43 with a patch series, 1 to 16 users - #2364

bench: community report, 4x RTX 3090 layer split, stock 0.1.44 against 0.1.43 with a patch series, 1 to 16 users#2364
havok2-htwo wants to merge 1 commit into
Niko1221:mainfrom
havok2-htwo:bench/2026-10-14-4x-rtx-3090

havok2-htwo commented Oct 14, 2026

Uh oh!

havok2-htwo commented Oct 18, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

havok2-htwo commented Oct 14, 2026

Uh oh!

havok2-htwo commented Oct 18, 2026

Uh oh!

Niko1221 commented Oct 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants